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Abstract 


A study  of  86  energy  management  and  control  systems  (EMCS)  was  made  to 
determine  users’  satisfaction  and  perceived  system  reliability.  Many  WC 
systems  reportedly  were  not  meeting  users’  expectations  for  energy,  dollar,  or 
manpower  savings.  Nearly  one-third  of  the  EMCS  users  in  this  study  were  not 
satisfied  with  their  system’s  performance.  Undependability  of  the  manufacturer 
and  proprietary  hardware  and  software  were  the  most  common  problems  reported. 
Factors  found  to  have  no  apparent  effect  on  user  satisfaction  were:  (l)  the 

type  of  facility  (with  the  exception  of  Hospitals)  in  which  the  EMCS  is 
installed,  (2)  which  manufacturer  supplied  the  system,  and  (3)  whether  or  not 
the  system  was  owned  by  the  Federal  Government. 

The  data  suggest  that  EMCS  maintenance  training,  in-house  maintenance 
capability,  and  reduced  dependence  on  the  system  manufacturer  may  help  improve 
user  satisfaction. 
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Foreword 


This  is  one  of  a series  of  reports  documenting  National  Bureau  of  Standards 
(NBS)  research  and  analysis  efforts  in  developing  energy  and  cost  data.  These 
data  support  the  Building  Energy  Conservation  Criteria  Program  sponsored  by  the 
Office  of  Buildings  and  Community  Systems,  U.S.  Department  of  Energy  (DOE).  The 
work  described  in  this  report  was  supported  by  DOE/NBS  Task  Order  No.  A 008-BCS 
under  Interagency  Agreement  No.  EA  77  A 01  6010. 
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1.  INTRODUCTION 


This  report  summarizes  the  findings  of  a study  made  to  determine  failures 
experienced  with  energy  management  and  control  systems  (EMCS) . The  goals  of 
this  study  were  to  determine  user's  satisfaction  with  their  EMCS,  and  to 
identify  common  problems  and  possible  cures,  through  interviews  with  system 
users. 

Because  no  scientific  method  was  used  in  the  process  of  selecting  the  study 
participants,  there  is  no  way  to  extrapolate  this  study  data  to  represent 
general  users  of  EMC  systems. 

1.1  EMCS  in  perspective 

A common  goal  of  all  EMC  systems  is  to  reduce  the  energy  costs 
associated  with  heating,  ventilating,  and  air-conditioning  (HVAC)*,  while 
meeting  a pre-selected  set  of  temperature  and  humidity  goals.  EMCS 
included  in  this  study  were  designed  for  use  in  applications  ranging  from 
small  buildings  with  as  few  as  50  monitor  and  control  points,  to  large 
multi-building  complexes  with  as  many  as  15,000  points.  These  systems  have 
environmental  and  control  sensor  data  brought  together  at  a central  control 
area  where  the  starting  and  stopping  of  HVAC  devices  can  be  performed. 

The  more  advanced  EMC  systems  also  include  one  or  more  computers  that 
turn  energy-consuming  devices  on  or  off,  control  dampers,  decide  which 
source  of  heating  or  cooling  is  most  efficient  for  present  conditions,  and 
shed  electrical  loads  to  keep  within  peak  electric  demand  limits. 

1.2  User  dissatisfaction  in  perspective 

The  information  presented  in  this  paper  is  not  based  on  precise 
failure  data;  it  contains  a degree  of  uncertainty  due  to  user  subjectivity. 
For  instance,  two  users  of  the  same  EMC  system  may  perceive  its  performance 
differently.  Another  factor  that  could  affect  integrity  of  the  data  is  the 
possible  reluctance  of  users  to  admit  failures  that  some  people  may 
construe  as  inadequate  operator  skills. 

No  previous  studies  about  EMCS  user  satisfaction  have  been  found.  However, 
two  guidelines  [l ,2]  that  provide  detailed  approaches  to  planning, 
specification,  and  installation  of  EMC  systems  are  available.  The  U.  S.  Navy 
guideline  [l],  referred  to  as  the  inter-agency  guide  spec.,  is  designed  to 
assist  all  government  agencies  with  their  EMCS  procurements.  The  Veterans 
Administration  guideline  [2]  is  a research  report  focusing  on  V.A.  hospital  EMC 
systems.  Although  neither  guideline's  primary  effort  is  the  discussion  of  EMCS 
problems,  they  may,  by  their  comprehensive  approach,  provide  a basis  for 
improving  performance  of  these  systems. 

Among  the  approaches  recommended  by  these  guidelines  is  the  decentralizing 
of  system  control.  This  "distributed  processing"  technique  has  the  potential 
for  reducing  the  load  on  a system's  main  computer  and  may  help  resolve 


*A  glossary  of  MCS  terms  is  located  at  the  end  of  this  report. 
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transmission  link  problems  uncovered  by  this  study.  The  "smart"  field  interface 
devices  (PID's)  recommended  are  able  to  continue  functioning  when  the  main 
computer  fails,  thus  improving  reliability. 

Since  distributed  processing  systems  tend  to  be  more  complex  and  costly 
than  non-distributed  EMC  systems,  only  multi-building  complexes  or  very  large 
building  are  likely  to  benefit  from  this  technology.  Some  of  these  new 
distributed  type  systems  were  about  to  come  into  use  as  this  study  was  in 
progress;  but  any  improvement  in  EMCS  performance  as  a result  of  this  new 
technology  will  have  to  be  determined  after  these  systems  have  been  operating 
for  some  time. 


2.  APPROACH 

A study  was  conducted  to  determine  failure  experiences  in  a large  number  of 
EMC  systems  installed  in  diverse  facilities.  By  contacting  users  close  to  the 
system  operation,  problems  causing  user  dissatisfaction  were  uncovered. 
Discussions  with  EMCS  users  centered  around  the  following  five  questions: 

What  are  or  have  been  the  major  problem  areas? 

Are  you  satisfied  with  your  EMCS  performance? 

Are  you  using  all  of  the  functions  purchased  with  the  system? 

Is  EMCS  maintenance  done  in-house  or  by  contract? 

Has  your  maintenance  team  had  EMCS  maintenance  training? 

Responses  to  these  questions  were  pursued  in  detail.  For  example,  a 
complaint  about  hardware  was  pursued  further  to  localize  it  to  the  computer, 
peripheral  devices,  or  sensor  failure.  Because  of  sometimes  conflicting 
responses,  careful  analysis  of  the  data  was  necessaiy.  "Yes,  I'm  satisfied  with 
my  system's  performance",  may  have  come  from  the  same  user  who  responded,  "We 
are  expanding  our  system  with  manufacturer  B's  equipment  due  to  all  the  problems 
we  had  with  manufacturer  A". 


Identification  of  the  problem  areas  and  analysis  of  the  study  data  follow. 
3.  PROBLEMS  IDENTIFIED 

Figure  1 shows  the  problems  that  were  identified  during  the  study  and  the 
number  of  times  each  problem  was  reported.  Since  the  complaints  are  not 
completely  independent,  the  following  examples  may  help  differentiate  the 
categories. 

The  most  common  problem  reported  was  undependability  of  the  manufacturer. 
Most  of  the  complaints  concerning  this  problem  centered  around  the  inability  of 
the  manufacturer  to  keep  a system  operational.  Inadequate  documentation,  poorly 
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Figure  1 - Problem  Identification  And  Extent 
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trained  service  people,  and  slow  response  on  service  calls  were  frequent 
complaints.  Other  complaints  included  slow  turn-around  for  software  changes  and 
having  to  make  repeated  calls  to  the  manufacturer  to  get  repairs  made. 

Proprietary  problems  were  reported  as  the  most  frustrating.  Skilled 
technicians  at  EMCS  sites  were  unable  to  repair  hardware  or  to  make  software 
changes  because  the  manufacturer  refused  to  supply  adequate  documentation.  In 
some  cases,  users  could  not  obtain  hardware  or  software  maintenance  training 
from  the  system  supplier  because  such  training  would  reveal  proprietary 
information. 

Users  mentioned  only  a few  specific  software  problems.  Among  then  was  the 
inability  of  the  software  to  perform  simple  data  gathering  or  averaging 
routines.  Other  software  problems  were  fairly  general,  such  as  intermittent 
failures  in  system  programs  causing  excessive  downtime,  or  the  inability  to 
bring  optimizing  programs  up  to  an  acceptable  performance  level. 

In  several  cases,  lightning  destroyed  major  electronic  assemblies  in  EMC 
systems.  Most  other  related  complaints  were  about  noise  effects  on  the  data 
transmission  link  which  resulted  in  significant  data  losses. 

Peripheral  device  and  other  hardware  problems  were  broadly  distributed. 
There  were  slightly  more  complaints  about  printers,  but  tape  and  disc  drives, 
loop  remotes,  and  data  gathering  panels  all  had  about  the  same  number  of  reports 
of  failures. 

Obtaining  spare  parts  was  also  reported  as  a frustrating  problem.  Some 
users  encountered  delays  of  several  weeks  in  getting  parts,  while  others 
encountered  delays  exceeding  six  months  before  getting  replacement  parts  for 
their  systems. 

Most  central  processing  unit  (CPU)  problems  were  intermittent  and,  like 
intermittent  software  problems,  the  source  was  hard  to  locate  and  that  caused 
much  system  downtime.  Only  a few  users  reported  specific  CPU  problems  such  as 
temperature  sensitivity. 

Complaints  about  sensor  problems  were  very  consistent:  all  were  about 

calibration  drift. 


4.  CHARACTERISTICS  OP  SYSTMS  WITH  HIGH  USER  SATISFACTION 

Data  indicated  that  there  were  three  major  differences  between  systems  with 
high  and  low  user  satisfaction.  The  data  in  Figure  2 show  user  satisfaction  may 
be  related  to  the  three  characteristics.  Many  more  users  who  have  had  EMCS 
maintenance  training  perceived  their  systems  as  reliable  than  unreliable.  Also, 
more  than  twice  as  many  users  who  have  in-house  maintenance  capability  judged 
their  system's  performance  to  be  reliable  than  unreliable. 

These  two  maintenance-related  characteristics  may  be  associated  with  the 
third,  which  is  the  use  of  optimizing  functions.  These  data  show  that  for  the 
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Figure  2 - Perceived  Unreliability  VS.  System  Characteristics 
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systems  that  have  optimizing  capability,  more  than  twice  as  many  users  reported 
that  their  systems  are  reliable  than  unreliable. 


5.  SCOPE:  HOW  EXTENSIVE  ARE  THE  PERCEIVED  UNRELIABILITY  PROBLEMS? 

Of  the  86  EMC  systems  included  in  this  study,  29$  of  the  users  were  not 
satisfied  with  their  system's  performance.  While  making  a comparison  of  the 
data,  several  areas  were  investigated.  These  were:  (l)  the  type  of  facility  the 
system  is  installed  in,  (2)  system  age,  (3)  manufacturer,  and  (4)  user  group. 
Analysis  of  these  data  follows. 

Perceived  Unreliability  vs.  Types  of  Facilities  - Figure  3 - The  data  presented 
on  this  graph  show  that  all  other  facilities  Have  EMCS  unreliability  problems 
greater  than  hospitals.  If  the  hospital  data  were  not  included  with  the  other 
study  data,  the  results  would  show  an  increase  from  29$  to  more  than  35$  of 
users  who  were  not  satisfied  with  their  system's  performance. 

Why  do  the  hospitals  have  systems  which  appear  more  reliable  than  those  in 
other  facilities?  Studying  age  profile,  manufacturer,  and  system  character- 
istics does  not  account  for  this  performance  difference.  One  factor  that 
contributed  to  hospital  reliability  is  the  V.A.  "Guidelines"  [2].  Increasing 
user  awareness  of  latent  problems  and  providing  an  effective  approach  for 
handling  EMCS  technology  in  the  hospitals  resulted  in  improved  reliability. 

Since  V.A.  hospitals  make  up  two-thirds  of  the  total  hospital  sample,  the  effect 
of  their  improved  reliability  affects  results  for  the  entire  group. 

Perceived  Unreliability  vs.  System  Age  - Figure  4 - The  overall  trend  of  this 
graph  shows  a general  increase  in  perceived  unreliability  with  increasing  age. 
This  seems  reasonable  since  failures  in  mechanical  parts  are  often  due  to  wear 
and  fatigue,  which  can  increase  with  age.  No  reason  has  been  found  to  explain 
the  break  in  trend  shown  by  the  highly  unreliable  4-year-old  systems  (1976 
vintage).  Also,  because  the  data  for  systems  that  are  less  than  two  years  old 
is  based  on  a very  small  sample,  those  results  could  be  misleading. 

Perceived  Unreliability  vs.  Manufacturers  - The  study  data  indicated  that  all 
manufacturers ' systems  have  a"  similar  perceived  reliability  record. 

Perceived  Unreliability  vs.  User  Groups  - To  compare  perceived  reliability 
inf o mati on the  system  data  were  separated  by  user  group  into  federal  military 
(25  systems),  federal  non-military  (41  systems),  and  non-federal  (20  systems). 
Comparison  of  these  data  shows  that  each  user  group  had  similar  perceived 
reliability  experiences. 

6.  SYSTEM  COMPARISONS 

During  the  study  two  differently  structured  EMCS  installations  were 
observed  to  compare  user  experiences  and  system  configurations.  Both  systems 
are  described  below  and  are  considered  to  be  reliable  by  the  system  operators. 
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Figure  3 - Perceived  Unreliability  VS.  Types  of  Facilities 
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The  first  system  is  a relatively  small  system  configured  like  many  in  the 
study  that  do  not  have  optimizing  functions.  It  has  remote  monitoring  and  data 
logging  of  control  points,  and  is  used  as  a manual  start/stop  center  for  single- 
huilding  HVAC  equipment.  The  automatic  mode  is  not  used  because  the  operators 
anticipate  heating  or  cooling  load  changes  and  adjust  the  system  manually.  They 
are  performing  functions  like  optimum  start/stop  and  load  shedding. 

One  factor  contributing  to  this  system's  perceived  reliability  is  in-house 
maintenance.  Many  replacement  parts  are  kept  on  hand  to  assist  in  repairing  by 
replacing  suspect  components.  The  system  manufacturer  is  only  occasionally 
called  for  repair  assistance,  while  the  system  operators  play  the  major  role  in 
making  this  system  reliable  and  effective.  It  should  be  noted,  however,  that 
this  system  does  not  operate  near  its  potential  for  automatic  control  with  the 
existing  equipment. 

The  second  system  is  one  of  the  largest  in  the  country,  and  is  used  as  a 
multiple-building  HVAC  control  system.  It  has  many  optimizing  functions 
including  optimum  start/stop,  load  shedding,  chiller/boiler  optimization,  and 
enthalpy  control.  Also,  a complete  back-up  computer  and  trend  log  are  available 
to  ensure  reliable  operation. 

Some  of  the  factors  that  users  say  contribute  to  this  system's  reliability 
are:  (l)  personnel  training,  (2)  English  language  man-machine  interfaces  at 

control  consoles,  (3)  electric  surge  protection  to  minimize  li^itning  problems, 
(4)  as-built  drawings  and  documentation,  and  (5)  maintenance  management  software 
to  keep  track  of  preventive  maintenance  schedules. 

Five  full-time  people  are  responsible  for  system  operation  during  each 
three-shift  work  day,  and  are  assisted  by  the  system  manufacturer  in  maintenance 
tasks.  Personnel  commitment,  capability  of  using  optimizing  functions,  and 
back-up  systems  are  all  factors  in  this  system's  high  reliability. 

This  comparison  of  two  dissimilar  systems  shows  that  user  satisfaction  or 
perceived  reliability  does  not  depend  on  system  size  or  the  use  of  automatic 
controls. 

7.  SUMMARY 

The  data  show  that  many  problems  do  exist  with  EMC  systems;  about  one-third 
of  the  system  users  in  this  study  reported  trouble  keeping  their  system 
operational.  Although  the  study  revealed  no  causative  factors  for  the  problems, 
some  associative  factors  have  been  identified.  Since  data  suggest  that  about 
half  of  all  identified  problems  are  related  to  excessive  dependence  on  the 
system  manufacturer,  reducing  that  dependence  could  be  a key  factor  in  improving 
EMCS  reliability. 

Analysis  of  system  characteristics  revealed  that  EMCS  maintenance  training 
and  in-house  maintenance  capabilities  were  found  much  more  often  with  those 
systems  that  users  perceived  to  be  reliable.  Also,  there  were  more  than  twice 
as  many  reports  of  reliable  systems  than  unreliable  systems  in  the  group  of 
users  whose  systems  have  optimizing  capability. 
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System  age  was  found  to  affect  reported  reliability  in  a logical  way;  that 
is,  perceived  unreliability  increases  with  increasing  age.  To  minimize  old  age 
effects  on  EMCS  performance  some  users  recommend  that  a preventive  maintenance 
program  be  implemented. 

From  other  system  data,  three  factors  were  found  to  have  no  apparent  effect 
on  perceived  reliability:  (l)  the  type  of  facility  (with  the  exception  of 

Hospitals)  in  which  the  system  is  installed,  (2)  which  manufacturer  supplied  the 
system,  and  (3)  whether  or  not  the  system  was  owned  by  the  Federal  Government. 
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Appendix  I 

Glossary  of  EMCS  Terms 


Control  points  - switches  through  which  remote  control  functions  are 
accomplished. 


Distributed  processing  - a technique  that  uses  microcomputer  controllers  at 

remote  sites  to  collect,  process,  and  store  sensor  data 
until  called  for  by  the  central  computer. 


Electric  demand  limit  - a predetermined  rate  of  electricity  use  beyond  which 

large  additional  fees  are  added  to  a customer’s 
electric  bill. 


Enthalpy  control  - an  optimizing  function  that  allows  use  of  outside  air  for 

cooling  when  total  heat  content  of  the  outside  air  is  less 
than  that  of  the  inside  air. 


HVAC  - heating,  ventilating,  and  air-conditioning. 


Intermittent  hardware/sof tware  problems  - those  problems  that  occur 

and  cannot  be  initiated  by  any  known  sequence 


randomly 
of  events. 


Load  shedding  - an  optimizing  function  that  cycles  electrical  loads  off  and  on  to 
keep  energy  use  below  the  electric  demand  limit. 


Optimizing  functions  - those  which  improve  efficiency  of  the  EMCS  by  automatic, 

real-time  adjustment  of  system  equipment. 


Optimum  start/stop  - an  optimizing  function  that  saves  energy  by  starting  or 

stopping  systems  after  evaluating  weather  conditions,  thermal 
inertia  of  the  buildings,  and  hours  of  occupancy. 


Trend  log  - a display  of  system  parameters  that  is  updated  frequently  to  establish 
temperature  and  energy  use  trends. 


Unreliability  - a broad  term  which  in  this  study  indicates  user  dissatisfaction 
with  EMCS  performance. 
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